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O Abstract: Methods and arrangements are disclosed for embedding and detecting a watermark in a cinema movie, such that the 
O watermaric can be detected in a copy made by a handheld video camera. The watermark embedder divides each image frame into 
^ two areas. A watermark bit *+l ' is embedded in a frame by increasing the luminance of the first part and decreasing the luminance 
Q of the second part. A watermark bit *-r is embedded by decreasing ihe luminance of the first part and increasing the luminance of 
^ the second part. It is achieved with the invention that the embedded watermark survives *de-flicker* operations that are often used 
to remove flicker caused by the difTerent frame rates of cinema projection equipment and consumer camcorders. 
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Watermaik embedding and detection of a motion image signal 



PCT/ffi2004/050010 



FIELD OF THE INVENTION 

The invention relates to a method and arrangement for embedding a 
watermaik in motion image signals such as movies projected in cinemas. The invention also 
relates to a method and arrangem^t for detecting a watermaik embedded in such motion 
5 image signals. 



BACKGROUND OF THE INVENTION 

Watermaik embedding is an important aspect of copy protection strategies. 

1 0 Although most copy protection schemes deal widi protection of electronically distributed 
contents (broadcasts^ storage media)» copy protection is also desired for movies shown in 
theaters. Nowadays, illegal copying of cinema material by means of a handheld video camera 
is already common practice. Al&ough the quality is usually low, the economical impact of 
iUegal VHS tapes, CD-Videos and DVDs can be enormous. For this reason, cinema owners 

15 are obliged to prevent the presence of video cameras on their premises. Not following this 
rule may be sanctioned with aban on the future availability of contoit In view hCTeof, it is 
envisioned to provide that a watermaik will be added during show time. The watermark is to 
identify the cinema, the pres^tation time, operator, etc. 

Robustness to geometric distortions is a key requirement for such watermaik 

20 embedding schemes. A handheld camera will not only seriously degrade the video by 
filtering (the optical path from the screen to the camera, transfer to tape, etc.) but also 
seriously geometrically distort tiie video (shifting, scaling, rotation, shearing, changes in 
perspective, etc.). In addition, these geometrical distortions can change from frame to frame. 

A prior-art method of embedding a watermark in cinema movies, which meets 

25 die robustness requirements, is disclosed in Jaap Haitsma and Ton Kalker: A Watermarking 
Scheme for Digital Cmema; Proceedings ICIP, Vol.2, 2001, pp. 487-489. The robustness to 
geometric distortions is achieved by exploiting only the temporal axis to embed &e 
watranark. The watermaik is a periodic pseudo-random sequence of watermark samples 
having two distinct values, e.g. * 1 ' and *-l One watermaik sample is embedded in each 



wo 2004/066626 PCT/IB2004/050010 

2 

image. The value *V is embedded in an image by increasing a global property (e.g. the mean 
luminance) of the image, the vahie '-1' is embedded by decreasing said global property. 

The prior-art watennaiic embedding method actually embeds flidcer. By 
embedding the same watmnaik sample in a number of consecutive images, the flicker is 
5 made imperceptible (tiie human eye is less sensitive to low-frequency flicker). 

Flicker of the recorded movie is also caused by a) the typical mismatch 
between the cinema projector's frame rate (24 frames per second) and the camcorder's frame 
rate (25 §)s for PAL, 29.97 fps for NTSC), and b) the difference between the two display 
scan formats (progressive vs. interlace). This kind of flidcer is so annoying that de-flickering 
10 tools have been made widely available to the public. For example, a de-flicker plug-in for the 
video capturing and processing application '*Virtualdub" has been found on the Internet 

A problem of the prior-art watennaik embedding scheme is that de-flicker 
tools also remove the embedded watermark. 



15 

OBJECT AND SUMMARY OF THE INVENTION 

It is an object of die invention to finrther improve the prior-art watermark 
embedding and detection method It is a particular object of die invention to provide a 
watermark embedding and detection scheme, which is robust to de-flickering operations. 

20 To this end, die method of embedding a watermark in amotion image signal 

according to the invention includes dividing each image into at least a first and a second 
image area. One value of a watermark san^le is embedded in an image by increasing the 
global property (e.g. the mean luminance) of its first area and decreasing the global property 
of its second area. The other value of the watennaik sarrq)le is embedded in the opposite way, 

25 Le. by decreasing the global property of the first image area and increasing the global 
property of the second image area. 

The invention exploits the insight that de-flickering tools remove flicker by 
adjusting the mean luminance of successive images to exhibit a low-pass character. The mean 
luminance is adjusted in practice by multiplying all pixels of an image by die same frictor. 

30 Because this operation does not affect the (sign of the) modifications applied to the distinct 
image areas, the watermark information will be retained. In view hereof the global property 
of an image area being modified to embed the watermark is die mean luminance of said 
image area. 
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In a preferred embodiment of the method, flie first and the second image area 
are the iqyper and lower half of an image. In general, there are more horizontal than vertical 
mov^ents in a movie. The horizontal movements influence the mean hmiinance values to a 
smaller extent 



BRIEF DESCRIPTION OF THE DRAWINGS 

Fig. 1 is a schematic diagram of a watermark embedder in accordance with the 

invention. 

10 Fig. 2 is a schematic diagram of a watemiatk detector in accordance with the 

invention. 

Fig. 3 is a schematic diagram of a correlation stage, which is an element of ttie 
watermark detector shown in Fig. 2. 

Fig. 4 shows graphs of the mean luminance values of an original image 
1 5 sequence and a watermarked image sequence. 



DESCRIPTION OF EMBODIMENTS 

Fig. 1 is a schematic diagram of a watermark embedder in accordance with the 

20 invention. The embedder receives a sequence of images or frames having a luminance F(njk) 
at spatial position n of frame k. The embedder further receives a watermark in die form of a 
pseudo-random sequence w(n) of length N, where w(n)e [-1, 1]. An appropriate value of N 
for this application is N=1024. The arrangement comprises a dividing stage 10, which divides 
each image into a first (e.g. iqpper half) area and a second (e.g. lower half) area. The 

25 luminance of said image areas is denoted FiOoJc) and F2(njk:), respectively. 

In the simplest embodiment of the watermark embedder, the seqiience w(n) is 
directly applied to embedding stages 11 and 12. In such an embodiment, embedding stage 11 
adds one applied watermark sample w(n) to every pixel of the first image area, whereas 
embedding stage 12 subtracts the same watemiark sample from every pixel of the second 

30 image area. Clipping is performed where necessary. The mean luminances of die first and 
second image areas are thus oppositely modulated by the watermark. 

Other examples of global image properties that can be modulated by the 
watermark are picture histograms (a list of relative firequ^cies of limiinance values in the 
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picture), or features derived therefrom such as high order moments (average of limiinance 
vahies to a power k). The mean hmiinance is a specific example of the latter (fc=l). 

Since the Human Visual Systran (HVS) is sensitive to flicker in low spatial 
firequencies, this simple embodiment may suffer from arti&cts in especially non-moving flat 
5 areas. These arti&cts are significantly reduced by lowering the flicker fi-equency of the 

watermark. This is performed by a repetition stage 13, which repeats each watermark sample 
during a predetermined number K of consecutive images. The same watermark sample is thus 
embedded in K consecutive frames. The watermark repeats itself every N=1024 firames. The 
watermark sample w(n) being embedded in frame k can be mathematically denoted by 
10 w(|k/KjmodN) , For simplicity, this e^qnession will hereinafter be abbreviated to w(k). 

The preferred embodiment of the embedder which is shown in Fig. 1 fiirther 
adapts the embedding depth in dependence iqpon the image contents. To this end, the 
embedder comprises multipliers 14 and 15, which multiply the watermaik sample w(k) by a 
local scaling &ctor CF,i(n^) and Cp^fejc), respectively. The local scaling fsLCtors are derived 
1 5 from the image contents by image analyzers 16 and 17, respectively. For example, they are 
large in moving textured parts of an area and small in non-moving flat parts. The outputs of 
the embedding stages 11 and 12 can be fonnulated as: 
Fw,i(n,k) = I^(n,k)+ CFj(n,k)w(k) 

Fw,2fek) = F2(n,k)-Cp^2(n,k)w(k) 

20 It will be appreciated that both embedding operations can be carried out in a time-sequential 
manner by a single processing circuit under q)propriate software control. 

The two image areas are subsequmtly combined by a combining stage 18 into 
a single watennaiked image Fw(n4c). 

Fig. 2 is a schematic diagram of a watermark detector in accordance with the 

25 invention. Although the original signal is available during detection, the detector does not 
utilize any knowledge about the original. The arrangement receives a watermarked sequence 
of images or firames having a limiinance Fw(n,k) at spatial position n of fi:ame k The detector 
comprises a dividing stage 20, which divides each image into a first (e.g. upper half) area and 
a second (e.g. lower half) area in a similar manner as dividing stage 10 (Fig. 1) of the 

30 embedder. The luminance of each image area is denoted Fw,i(|Lk) and Fw^(njc), respectively. 
For each image area, the detector further includes a mean luminance computing circuit 
21, 22, which computes the mean luminance values fw,i(k) and fw;j(k) (or other global 
property, if applicable) of the respective image areas in accordance with: 
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fw,iW=^Il^w.i(S,k) 
n 

In practice, the mean luminance vahies of a movie exhibit a low-pass nature as 
a function of the frame number k (Le. as a function of time). Hie detector estimates the mean 
luminance values of the original (unwatemiaiked) movie by low-pass filtering (25, 27) tiie 
5 respective mean luminance values fw.i(k) and fw^Oc). Estimations of the mean lununance 
modifications as introduced by the embedder are subsequentiy obtained by subtracting 
(26, 27) the low-pass filtered mean values from the imfiltered mean luminance values. The 
detector estimates the embedded watennaik sample by subtracting (23) both estimations, 
followed by a sign operation (29). The estimated watermark sample being embedded in frame 

10 k is denoted v(k). 

The arrangement thus generates a sequence of estimated watermark samples. 
In a correlation stage 3, the estimated sequence of watermark samples is correlated wilh the 
watennaik being looked for. The detector receives tiiis watermark being looked for in the 
form of a pseudo-random sequence w(n) of length N, where w(n)e [-1, 1]. The detector 

15 comprises a repetition stage 24, which is identical to same repetition stage 13 of the 

embedder. The repetition stage rq>eats each watermark sample during K consecutive images. 
The watermark repeats itself every N=1Q24 frames. The watermark samples being appUed to 
the correlation stage 3 for frame are denoted w(k). Again, w(k) is an abbreviation for tiie 
mathematically more correct expression w(Lk/KjmodN). 

20 It should be noted that the low-pass filter / subtracter combinations 25,26 and 

27^, as well as the sign operation 29 are optional. 

Fig. 3 is a schematic diagram of the correlation stage 3. The estimated 
watermark samples of successive images are distributed to K bufiTers 31, 32, where, as 
described above, K is the number of consecutive images in which the same watermark 

25 sample is embedded. Each buffer stores N estimated watermark samples (or N computed 

mean luminance values, or N estimated mean luminance modification values). Topical values 
of the watermaric length N and the frames per watermark sample K are 1024 and 5, 
respectively. Accordingly, the first buffer 31 contains estimated watermark samples v(l), 
v(6), v(l 1), .., the second buffer 32 contains v(2), v(7), v(12), .., etc. This implies that the 

30 granularity of watermark detection is approximately 3 minutes and 25 seconds for PAL 
video. 
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The watennark is detected by detennining the similarity of the contents of 
each buffer with the reference watennaik w(n) being looked for. Each watermark can 
identify, for instance, one theatre. A well-known example of similarity is cross-correlation, 
but other measures are possible. The contents of each buffer are cross-correlated with the 
5 reference watermaric in respective correlators 33, 34, ... The correlation is preferably 

performed using Symmetrical Phase Only Matched Filtering (SPOMF). For a description of 
SPOMF, reference is made to International Patent plication WO 99/45706. In said 
document, tiie correlation is performed in the two-dimensional spatial domain. Blocks of 
NxN image pixels are correlated with an NxN reference watermark. The result of the 

10 SPOMF operation is an NxN pattern of correlation values exhibiting one or more peaks if a 
watermark has hem embedded. 

The K correlators 33, 34, operate in die one-dimensional time domain. The 
output of each correlator is a series of N correlation values which is stored m a corresponding 
one of K buffers 35, 36, A peak detector 37 searches the highest correlation value in the K 

15 buffers, and ^plies said peak value to a threshold circuit 38. If the peak value of at least one 
of the buffers is larger than a given dureshold value, it is decided that 1b& watennark is 
present Otherwise, tibie content will be classified as not watermarked. 

A payload can be encoded in the signal by embedding shifted versions of the 
watennark w(n) in a manner similar to the one disclosed in International Patent Application 

20 WO-A-99/45705. It should further be noted that, although parallel correlators are shown in 
Fig. 3, it may be advantageous to carry out the respective operations in a time-sequential 
manner. 

In order to describe tiie invention in even more details, a mathematical 
analysis of the prior-art watermarking scheme, an analysis of tiie piiblicly available de- 

25 flickering tool, and die operation of the watermarking scheme in accordance with the 
invention will now be given. 

The watomark w is a periodic pseudo-random sequence containing only * 1 ' 
and *-r sample values with period M. A watermaric sample w(n) is embedded in K 
consecutive frames k, k+1,.., k+K-1. By embedding one watermark sample in K consecutive 

30 frames, the fi^uency of the flickering due to the anbedding is decreased. A * T is embedded 
in an image by increasing the luminance value of each pixel with a value CF(n,k). A 1' is 
embedded by decreasing the lununance value of each pixel with CpCn^c). Herein, n is the 
spatial coordinate of a pixel within frame k. More mathematically, we have: 
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rF(n,k)+Cp(n,k) ifwLkn^J=l 
^ lF(n,k)-CF(n,k) ifw[kyTj=-l 

where F is the fiame to be embedded and Fw is the embedded frame. The change Cf is chosen 
to be such that the watermark is not visible, and therefore depends on F. In [2], a texture 
detector and a motion detector are used to determine Cf- As a result, the mean lummance 
5 vahies of the watermarked video fw 

fw(k)=^ZF^(n,k), 
n 

with N being the number of pixels per frame, will exhibit a change with respect to flie 
original mean luminance values 

f^(k) = f(k)+CF(k)w(Lk/Tj) (1) 
10 Herein, Cf is the local depth of the watermark w, which is directly related to Cp: 

<^FW=:^ECp(n,k) 
n 

Fig. 4 shows, at (a), a graph of the mean luminance values of an original 
sequence and, at (b), a graph of the mean luminance values of an embedded sequence to 
visualize the watermark embedding concept 

1 5 Due to the watermark embedding, the mean luminance values will decrease or 

increase with respect to the original mean luminance values in time. See Eq, (1). In practice, 
die mean luminance values of a movie exhibit a low-pass nature. Therefore, Uxo detector 
estimates these luminance values of the original unwatennarked movie by low-pass filtering 
die mean luminance values of the watermarked movie fw. Hie detector estimates the 

20 watermark v by subtracting these low-pass filtered means fixym die mean luminance values of 
the watermarked movie fw, followed by a sign operation. More mathematically, 

v(k) =sign{f^(k)-(f^ OgXk)} (2) 
where ® denotes a (one-dimensional) convolution, and g is a low-pass filter. Since a 
watermark sample is ^bedded in K consecutive frames, this operation yields K estimates 

25 W| of die watermark w: 

w,(k) = v(l+kK),0<l<K (3) 

Each of these K estimated watermarks Wj is correlated with the original 
watermark w. If the absolute correlation value d| = |(w,wi)| is larger than a threshold value, 
it is decided that the video sequence is watemiaiked. 
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A movie is projected progressively at a frame rate of 24 frames per second 
(fps), however, a standard video camera records at 25 (PAL) or 29.97 ^ (NTSC) 
interlaced Due to this interlacing, the Imninance will not be Ae same throughout recording 
of one frame, as the shutter may just be opening or just be closing. Since the video camera 
5 and the projector are not synchronized, this problem is difficult to be solved for a camera 
man. In addition, since the frame rates of the projector and the video camera are not matched, 
a similar problem reveals itself; at some points in time, a frame is recorded when the shutter 
is half-open or even fully shut The result of these mismatches is a flickering in the recorded 
movie. 

10 A Me-flicker* phjg-in for Virtualdub can be found on the Internet This plug-in 

removes flie flickering in four steps: 

In a first pass, it calculates tiie mean luminance values f^ of die movie; 

Subsequently, it filters these means f ^ with a low-pass filter h (default is a sunple averagmg 

filter of length 12); 

15 Thea it calculates fectors P(k) between the origmal means fy,(k) and die filtered means for 
each firame: 

P(,),(fw^«>hXk) 
fw(k) 

In a second pass, the luminance value of each pixel in frame k is multiplied by the 
corresponding fector P(k) rounded to ttie nearest integer, and clipped if it exceeds the 
20 maYiTtiinn luminance value 255. 

Note that P(k) is non-negative for all k, since f^ (k) > 0 and h is a low-pass 
filter. If we ignore the roimduig and the clipping for the moment, the result of these 
multiplications in the last step is that die means of the new constructed video fw,deflic 

resembles die low-pass filtered means of : 

25 fw.deflic =P(k)f(k) = (fw «>h)(k) 

Perceptually, this means that the new constructed video exhibits less 
flickering, because flickering can be seen as a high-frequency component in die mean 
luminance values of the frames, which are now filtered out. 

Unfortunately, the watermarking scheme is actually a flickering, although 
30 imperceptible. As a direct consequence, this 'de-flicker* plug-in removes the watermark. The 
watermark embedding scheme must thus be modified in such a way that it is robust to de- 
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flickering. This is all the more true as this 'de-flicker' tool is widely used to iindo the pirate 
copies from ttie flickering. 

To this end, each frame is divided into two parts (e.g. lefl/right or top/bottom), 
and the watennaik sample is embedded in these parts in an opposite way. To embed a 
5 watermark sample, the mean luminance of one part is increased and the mean luminance of 
die other part is decreased. Consequently, the mean luminance values fw of the watermarked 
movie now consist of two parts 

fw(k)=fw4(k)+fw,2(k) 
with(c£Eq. (1)) 
10 fw4(k) = fi(k)+CF,i(k)w(Lk/Tj) and 

fw.2(k) = f2(k)-CF,2(k)w(Lk/Tj) (4) 

After c^turing witfi a camera, the 'de-flicker' tool removes the flickering by 
low-pass filtering the mean luminance values f^ : 

fw,demc(k)=P(k)fw(k)|fw4(k)+W(*^J 

1 5 The detection of the watermark for the modified watermarking scheme is 

similar to tiie detection me&od described above. First, the detector estimates the luminance 
values of the original xmwatermarked movie for both parts by low-pass filtering the mean 
luminance values of both parts. Then it subtracts the result of both operations from the 
luminance values of the corresponding parts. Finally, it makes an estimate of the watermark 

20 V by subtraction followed by a sign operation (cf. Eq, (2)): 

v(k) = sign{(f w^i^defic 00 " (fw.l,deflic ® gXk) - (f w,2,defic OO " (f wAdeflic ®g)(k))} (5) 

The K estimations W) (see Eq. (3)) are obtained in a similar way and 
correlated with tiie watermark w. 

The embedded watermark survives the de-flicker operation because the de» 

25 flicker tool multiplies all the pixels of an image by the same &ctor P(k), thereby leaving the 
luminance differences between flie two image areas substantially intact The effect of the 
invention can also be explained more mathematically. It is assumed that the original 
unwatermarked movie exhibits a low-pass nature. After capturing of the movie with a 
camera, the mean luminance values of the watermarked parts f^ j and f^2 exhibit a 

30 flickering 

f^j(k) = 7(k)fw,i(k)and 
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fw,2(k) = 7(k)fw^(k) 

Herein, Y(k)>0 corresponds to tiie change in the mean luminance vahie (the flickering) of 
frame k. The 'de-flicker* plug-in removes this flickering by low-pass filtering the mean 
luminance values 
5 fw.demc(k)=P(k)fw(k)=P(k)7(k)[f(k)+t^^^ 
From this expression it follows that 

f(k) 

P^^^^" f(k)+jcp4(k)-CF^(k)]w(Lk/Tj) 
Since in practice {cp j (k) ~ cf^2 (k)]w ([k / T J) is relatively small compared to J^), we can 
approximate P(k)Y(k) by 1. By using feis approximation, we see that 
10 fw,Udeflic(k) = P(k)7(k)fw4 (k) - fw,l(k) = fi(k) +CF^i(k)w(Lk/Tj) 

For the other part, we obtain a similar result 

fw,2.deflic(k) - f2(k)-CF,2(k)w(Lk/Tj) 
Usmg tiiese results, we finally obtain the following e^ession for v (see Eq. (5)) 
v(k) « sign^i(k)-f2(k)+[cF,i(k)+CF,2(k)Jwlk/Tj)-[fi(k)-f2(k)^^ 
= sign|cF4(k)+CF,2(k)]w(Lk/Tj)}==sign{w(Lk/Tj)} 

1 5 where it is assumed tiiat the low-pass filter g completely filters out the watermaric Note that 
cfj (k) + cf^ (k) does not influence fhe sign of the e^qjression, because it is non-negative for 
all k. It can be seen from flus expression that the watermark mdeed survives the 'de-flicker* 
operation after the modificatiorL 

Methods and arrangements are disclosed for embedding and detecting a 

20 watermark in a cinema movie, suchthatthe watermark can be detected in a copy made by a 
handheld video camera. The watermark embedder divides each image frame into two areas. 
A watermark bit *+!' is embedded in a frame by increasing the luminance of the first part and 
decreasing the luminance of the second part A watermark bit '—1 ' is embedded by 
decreasing the luminance of the first part and increasing the luminance of the second part It 

25 is achieved with the invention that the embedded watermark survives * de-flicker' operations 
that are often used to remove flicker caused by the different fiame rates of cinema projection 
equipment and consumer camcorders. 
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CLAIMS: 



1. A metliod of embedding a watermaik in a motion image signal, the method 
comprising the steps of: 

- representing said watermark by a sequence of watermark samples each having a first or a 
second value; 

5 — dividing an image of said motion image signal into at least a first and a second image 
area; 

— determining a global property of the first and the second image area; 

- modifying said image to increase tiie global property of its first area and decrease tiie 
global property of its second area for embedding the first value of a watermark sample 

10 into said image, and to decrease the global property of its first area and increase the 
global property of its second area for embedding the second vahie of said watermark 
sample into said image. 

2. A method as claimed in claim 1, wherein said global property is the mean 
1 S luminance value of the respective image area. 

3. A method as claimed far claim 1, wherein said modifying step comprises 
modifying series of consecutive images in accordance wiA the same watermark sample. 

20 4. A method as claimed in claim 1, wherein said first and second image areas are 

the iq>per and lower of an image halves, respectively. 

5. A method as claimed in claim 1, wherein said first and second image areas are 
Ae left and right of an image halves, respectively. 

25 

6. An arrangement for embedding a watermark in a motion image signal, the 
arrangement comprising: 

— means for representing said watermark by a sequence of watermaric samples each having 
a first or a second value; 
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- means for dividing an image of said motion image signal into at least a first and a second 
image area; 

- means for detennining a global property of the first and the second image area; 

- image modifying means being arranged to increase the global property of flie first image 
S area and decrease the global property of the second image area in response to &e first 

value of a watermark sample to be embedded into said image» and to decrease die global 
property of the first image area and increase tiie global property of the image second area 
in response to embedding die second value of a watermark sample to be embedded into 
said image. 

10 

7. A metiiod of detecting a watermark in a watermarked motion image host 

signal, the method comprising the steps of. . 

- dividing each image of said host signal into at least a first and a second image area; 
~ determining a global property of die first and the second image area; 

15 - computing, for each of a series of images, the difference betwerai the global property of 
the first and the second image area; 

- correlating, for said series of images, the respective dififerences widi the watermark to be 
detected. 

20 8. A method as claimed in claim 7, wherein said global property is the mean 

luminance value of the respective image area 



9. A method as claimed in claim 7, further including die step of subtracting from 

the series of global properties a low-pass filtered version thereof, and applying die correlating 
25 step to the subtracted signal. 



10, A mediod as claimed in claim 9, fiirther including die step of determining the 

sign of said subtracted signal, and applying die correlating step to said sign. 
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